<html>
<head><meta charset="utf-8"><title>Switchig parser from splitting tokens to gluing tokens · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Switchig.20parser.20from.20splitting.20tokens.20to.20gluing.20tokens.html">Switchig parser from splitting tokens to gluing tokens</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="172707383"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Switchig%20parser%20from%20splitting%20tokens%20to%20gluing%20tokens/near/172707383" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Switchig.20parser.20from.20splitting.20tokens.20to.20gluing.20tokens.html#172707383">(Aug 07 2019 at 19:24)</a>:</h4>
<p>My understanding is that we want to switch the parser from "composite tokens are always glued" to "composite token are always split" model, to mesh better with proc_macro token trees. <span class="user-mention" data-user-id="123856">@Vadim Petrochenkov</span> , <span class="user-mention" data-user-id="119009">@eddyb</span> is there any existing issue for it? Should I just create a new one? I am not sure what's the best way to proccede here: seems like I should just change everything in one go?</p>



<a name="172707410"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Switchig%20parser%20from%20splitting%20tokens%20to%20gluing%20tokens/near/172707410" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Switchig.20parser.20from.20splitting.20tokens.20to.20gluing.20tokens.html#172707410">(Aug 07 2019 at 19:24)</a>:</h4>
<p>Also, do we want to kill the glued tokens themselves?</p>



<a name="172709239"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Switchig%20parser%20from%20splitting%20tokens%20to%20gluing%20tokens/near/172709239" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Switchig.20parser.20from.20splitting.20tokens.20to.20gluing.20tokens.html#172709239">(Aug 07 2019 at 19:48)</a>:</h4>
<blockquote>
<p>is there any existing issue for it?</p>
</blockquote>
<p>I don't think so.</p>



<a name="172709265"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Switchig%20parser%20from%20splitting%20tokens%20to%20gluing%20tokens/near/172709265" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Switchig.20parser.20from.20splitting.20tokens.20to.20gluing.20tokens.html#172709265">(Aug 07 2019 at 19:49)</a>:</h4>
<blockquote>
<p>seems like I should just change everything in one go?</p>
</blockquote>



<a name="172709394"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Switchig%20parser%20from%20splitting%20tokens%20to%20gluing%20tokens/near/172709394" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Switchig.20parser.20from.20splitting.20tokens.20to.20gluing.20tokens.html#172709394">(Aug 07 2019 at 19:50)</a>:</h4>
<p>Perhaps it can be split.<br>
E.g. the primary parsing (<code>Parser</code> methods in <code>parser.rs</code>) can glue tokens on <code>bump</code> and still work with old-style tokens for some time, while everything else is converted.</p>



<a name="172709407"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Switchig%20parser%20from%20splitting%20tokens%20to%20gluing%20tokens/near/172709407" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Switchig.20parser.20from.20splitting.20tokens.20to.20gluing.20tokens.html#172709407">(Aug 07 2019 at 19:50)</a>:</h4>
<blockquote>
<p>Also, do we want to kill the glued tokens themselves?</p>
</blockquote>



<a name="172709443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Switchig%20parser%20from%20splitting%20tokens%20to%20gluing%20tokens/near/172709443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Switchig.20parser.20from.20splitting.20tokens.20to.20gluing.20tokens.html#172709443">(Aug 07 2019 at 19:50)</a>:</h4>
<p>The knowledge about glued tokens have to exist at least in some form because the <code>tt</code> matcher accepts a glued token.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>